Detecting encoded signals under adverse lighting conditions using adaptive signal detection

ABSTRACT

Signal detection adaptable to accommodate various lighting conditions, and to detect embedded signals according to anticipated lighting conditions likely to be present during image capture. A portable apparatus comprising: a camera for capturing data representing imagery or video; a touchscreen display; means for communicating and receiving data; means for obtaining data representing imagery or video, the imagery or video having been captured with said camera; means for obtaining lighting information associated with image capture of the imagery or video, in which the lighting information is associated with a light level or color temperature; and means for processing the data representing imagery or video to determine whether a signal is encoded therein, in which said means for processing utilizes obtained lighting information to determine whether the signal is encoded in the data representing imagery or video. Additional disclosure, combinations and claims are also provided.

RELATED APPLICATION DATA

This application is a continuation of U.S. application Ser. No.14/869,365, filed Sep. 29, 2015 (now U.S. Pat. No. 9,940,684) which is acontinuation of U.S. application Ser. No. 13/165,564, filed Jun. 21,2011 (now U.S. Pat. No. 9,147,222) which claims benefit of U.S.Provisional Application No. 61/357,864, filed Jun. 23, 2010.

This application is related to U.S. patent application Ser. No.12/634,505, filed Dec. 9, 2009 (published as US 2010-0150396 A1) andSer. No. 12/337,029, filed Dec. 17, 2008 (published as US 2010-0150434A1).

Each of the above patent documents is hereby incorporated by referencein its entirety.

TECHNICAL FIELD

The present disclosure relates generally to encoded signals,steganographic data hiding and digital watermarking. In some areas, thepresent disclosure relates to a dynamic signal detector that adaptsoperation based on lighting information. In other areas, the presentdisclosure relates to embedding signals in anticipation of the type oflikely lighting conditions present during image capture.

BACKGROUND AND SUMMARY

The term “steganography” generally infers data hiding. One form of datahiding includes digital watermarking. Digital watermarking is a processfor modifying media content to embedded a machine-readable (ormachine-detectable) signal or code into the media content. For thepurposes of this application, the data may be modified such that theembedded code or signal is imperceptible or nearly imperceptible to auser, yet may be detected through an automated detection process. Mostcommonly, digital watermarking is applied to media content such asimages, audio signals, and video signals.

Digital watermarking systems may include two primary components: anembedding component that embeds a watermark in media content, and areading component that detects and reads an embedded watermark. Theembedding component (or “embedder” or “encoder”) may embed a watermarkby altering data samples representing the media content in the spatial,temporal or some other domain (e.g., Fourier, Discrete Cosine or Wavelettransform domains). The reading component (or “reader” or “decoder”)analyzes target content to detect whether a watermark is present. Inapplications where the watermark encodes information (e.g., a message orpayload), the reader may extract this information from a detectedwatermark.

A watermark embedding process may convert a message, signal or payloadinto a watermark signal. The embedding process may then combines thewatermark signal with media content and possibly another signals (e.g.,an orientation pattern or synchronization signal) to create watermarkedmedia content. The process of combining the watermark signal with themedia content may be a linear or non-linear function. The watermarksignal may be applied by modulating or altering signal samples in aspatial, temporal or some other transform domain.

A watermark encoder may analyze and selectively adjust media content togive it attributes that correspond to the desired message symbol orsymbols to be encoded. There are many signal attributes that may encodea message symbol, such as a positive or negative polarity of signalsamples or a set of samples, a given parity (odd or even), a givendifference value or polarity of the difference between signal samples(e.g., a difference between selected spatial intensity values ortransform coefficients), a given distance value between watermarks, agiven phase or phase offset between different watermark components, amodulation of the phase of the host signal, a modulation of frequencycoefficients of the host signal, a given frequency pattern, a givenquantizer (e.g., in Quantization Index Modulation) etc.

The present assignee's work in steganography, data hiding, digitalwatermarking and signal detection is reflected, e.g., in U.S. Pat. Nos.7,072,487; 6,947,571; 6,912,295; 6,891,959; 6,763,123; 6,718,046;6,614,914; 6,590,996; 6,522,769; 6,408,082; 6,122,403 and 5,862,260, andin published specifications WO 9953428 and WO 0007356 (corresponding toU.S. Pat. Nos. 6,449,377 and 6,345,104), and in published U.S. PatentApplication No. US 2008-0298632 A1. Each of the patent documentsmentioned in this paragraph is hereby incorporated by reference in itsentirety. Of course, a great many other approaches are familiar to thoseskilled in the art. The artisan is presumed to be familiar with a fullrange of literature concerning steganography, data hiding and digitalwatermarking.

A so-called “fingerprint” may include characteristic features used toidentify a video or image. Such characteristic features can be derived,calculated or extracted from an image or video itself. Some suchcharacteristic features may include, e.g., frequency domain features,peaks, power characterizations, amplitude values, statistical features,key frame analysis, color, motion changes during a video sequence,and/or others. Characteristic features (e.g., one or more fingerprints)of artwork, or a portion thereof, can be distilled into a set ofnumbers, or features, which can be stored in a database, and latermatched against unknown works to identify the same. A fingerprint alsocan be used to link to or access remote data. Example image and videofingerprinting techniques are detailed, e.g., in U.S. Pat. Nos.7,930,546, 7,289,643, and 7,020,304 (Digimarc); U.S. Pat. No. 7,486,827(Seiko-Epson); 20070253594 (Vobile); 20080317278 (Thomson); and20020044659 (NEC). Each of the patent documents mentioned in thisparagraph is hereby incorporated by reference in its entirety.

One possible combination of the present disclosure includes a methodcomprising: obtaining data representing imagery or video, the imagery orvideo having been captured with a camera; obtaining lighting informationassociated with image capture of the imagery or video; based on thelighting information, adapting a signal detection process; and using aprogrammed electronic processor, analyzing the data to determine whethera signal is encoded therein, said act of analyzing utilizes an adaptedsignal detection process. Of course, different combinations and claimsare provided too.

Another combination includes a cell phone having: a camera for capturingimagery or video; and one or more electronic processors. The one or moreelectronic processors are programmed for: obtaining data representingimagery or video captured with the camera; obtaining lightinginformation associated with the captured imagery or video; based on thelighting information, adapting a signal detection process; analyzing thedata to determine whether a signal is encoded therein, said analyzingutilizes the adapted signal detection process.

Still another combination includes a method comprising: obtaininglighting information associated with image or video capture; adapting asignal detection process to deemphasize signal contribution of the bluechannel when the lighting information is associated with incandescentlighting; and using an electronic processor programmed with the adaptedsignal detection process, analyzing image or video data to determinewhether a signal is encoded therein.

Yet another combination includes a method comprising: obtaining datarepresenting captured imagery or video, the imagery or video having beencaptured with a camera; using a programmed electronic processor,analyzing the data to determine image statistics, the image statisticsidentifying a first region and a second region, in which the firstregion and the second region include different lighting characteristics;adapting a signal detector in a first manner for analyzing data in thefirst region, and adapting a signal detector in a second, differentmanner for analyzing data in the second region.

Another combination includes a method comprising: obtaining lightinginformation associated with image or video capture; adapting a signalembedding process to embed a digital watermark signal at a uniformembedding strength across two color channels when the lightinginformation is associated with incandescent lighting; and using anelectronic processor programmed with the adapted signal embeddingprocess, embedding an image or video to include the digital watermarksignal across the two color channels.

Further combinations, aspects, features and advantages will become evenmore apparent with reference to the following detailed description andaccompanying drawing.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram illustrating an exemplary cell phone.

FIG. 2 is a diagram showing the spectra of incandescent light versusdaylight/cool white fluorescent lighting.

FIG. 3 is a diagram showing detection rates of a 1:1 embedding channelratio vs. a 2:1 embedding channel ratio (blue/yellow:red/green) usingthe same detection weightings.

FIG. 4 is a diagram showing detection rates with incandescent lowlighting, using a 1:1 embedding channel ratio, and graphed showingdifferent color channel weightings.

FIG. 5 is a diagram showing detection rates with cool white lighting,using a 1:1 embedding channel ratio, and graphed showing different colorchannel weightings.

FIG. 6 is a diagram showing one example of a dynamic signal detector, inwhich detection is adapted based on lighting information.

FIG. 7 illustrates a smart phone camera casting a shadow on an object.

FIG. 8a and FIG. 8b illustrate apparent movement between a shadow and anobject being imaged.

DETAILED DESCRIPTION

We have found ways to improve detection of signals hidden in variouscolor channels, under sometimes adverse lighting conditions. Forexample, we have improved detection of encoded signals hidden in two ormore chrominance directions by adapting detection based on lightingconditions. These improvements include a consideration of variouslighting information (e.g., in terms of color temperature and/or lightintensity) and signal encoding techniques.

An exemplary use scenario operates on a color image or video including asignal encoded therein. One type of encoding encodes digitalwatermarking in a plurality of color channels. For example, the colorimage or video may be represented in the industry standard luminance andchrominance color space called “Lab” (for Lightness (or luminance), plus‘a’ and ‘b’ color channels). Of course, the present disclosure willapply to and work with other color schemes and techniques as well. Forexample, alternative luminance and chrominance color schemes include“Yuv” (Y=luma, and ‘u’ and ‘v’ represent chrominance channels) and “Ycc”(also a dual chrominance space representation).

In a case where a media signal includes (or may be represented by) atleast two chrominance channels, a watermark embedder may insert the samedigital watermark signal in both the ‘a’ color direction and ‘b’ colordirection. In one example the ‘a’ color direction represents a“blue/yellow” color direction, and the ‘b’ color direction represents a“red/green” color direction. This type embedding can be performed inparallel (if using two or more encoders) or serial (if using oneencoder). The watermark embedder may vary the gain (or signal strength)of the watermark signal in the ‘a’ and ‘b’ channels to achieve improvedhiding of the watermark signal. For example, the ‘a’ channel may have awatermark signal embedded with signal strength (or intensity) that isgreater or less than the watermark signal in the ‘b’ channel. A HumanVisual System response indicates that about twice the watermark signalstrength can be embedded in the blue/yellow channel as the red greenchannel and still achieve favorable (e.g., equalized) visibility.Alternatively, the watermark signal may be embedded with the samestrength in both the ‘a’ and ‘b’ channels. Regardless of the watermarkembedding strength, watermark signal polarity is preferably inverted inthe ‘b’ color plane relative to the ‘a’ color plane. The inverted signalpolarity is represented by a minus (“−”) sign in equation 2.WMa=a(channel)+wm  (1)WMb=b(channel)−wm  (2)WMa is a watermarked ‘a’ channel, WMb is a watermarked ‘b’ channel, andwm represents a watermark signal. A watermarked color image or video(including L and WMb and WMa) can be provided, e.g., for printing,digital transfer or viewing. When printing this type of watermarking innewspaper print the watermark signal is mainly in yellow and magentacolors. Capture, e.g., with a cell phone, of such newspaper printutilizes at least the blue and green channels under white fluorescentlighting.

An encoded signal may include a message or payload having, e.g., a linkto a remote computer resource, metadata or ownership information. Thecolor image or video is rendered (e.g., printed, distributed ordisplayed). A user, e.g., equipped with a camera enabled cell phone,captures an image of an encoded color image or video with her cell phonecamera. The captured image data is analyzed by a signal detector(embedded in the cell phone) to recover the message or payload. Thepresent disclosure provides methods and apparatus to improve thedetection of such encoded signals.

While the present disclosure focuses on detection of encoded signalswith a handheld device (e.g., camera equipped cell phone), other devicesmay be used as well. For example, digital cameras, scanners, webcameras, etc. may include or communicate with a detector. Thus,reference to a cell phone should not limit this disclosure.

FIG. 1 shown an exemplary cell phone, including, e.g., elements such asa microphone, a camera, a processor, a display/touchscreen, a physicaluser interface, a RF transceiver, location module (e.g., GPS), networkadaptor and memory. The memory may store operating system software, userinterface software, signal detector software, other functional softwaremodules, etc. Of course, cell phones including more or less featureswill also benefit from the present disclosure.

Signal noise in a captured image may be dependent on illuminationconditions. Generally, the lower the light level during image capture,the more noise that will be present in a captured image. This noiseappears to include random noise from a camera sensor. Noise isparticularly severe for cell phones which have small image sensors andcapture less light and, thus, have higher noise levels in capturedimagery. Noise can be further amplified in color cameras whereadditional amplification is sometimes used for the blue color channel incomparison to the green or red channels. Thus, in such cases, there isadditional observable noise in the blue channel.

Because of the association of noise and lighting conditions, we achievefavorable detection results when lighting conditions are considered inthe detection process, e.g., lighting level (e.g., lux level) and/orcolor temperature (e.g., type of lighting).

Before we discuss specific detection processes, we offer a few commentson lighting sources and color temperature.

The spectral power distribution of a cool white fluorescent bulb (“CWF”)is similar to daylight and reasonably balanced across the spectrum (seeFIG. 2). Under these conditions, and with a signal encoded with a biastoward the blue channel (e.g., an embedding ratio of 2:1 across theblue/yellow and red/green channels), the encoded signal has most of thesignal energy in blue and a signal detector reads the signal well withthis lighting. An embedding ratio of 2:1 may indicate that there istwice the signal (e.g., in terms of strength, intensity or magnitude) inthe blue/yellow channel relative to the red/green channel. An embeddingratio of 2:1 under these lighting conditions yields favorableimperceptible encoding.

In contrast to CWF lighting, incandescent lighting has an irregularpower distribution across the spectrum as shown in FIG. 2. Visible bluelight has a wavelength of about 400 nm, while visible red light has awavelength of about 650 nm. As a result, the blue channel only has aboutone tenth of the light as in the red channel under incandescentlighting. As mentioned above, lower light levels during image captureare prone to introduce more noise in captured imagery. Additionally, theblue channel is more prone to sensor noise amplification as well. Thus,under incandescent lighting, the blue channel as captured with a cellphone camera may be a noisy channel.

As mentioned above, lighting conditions are preferably considered in adetection process, e.g., lighting level (e.g., lux level) and/or colortemperature (e.g., type of lighting).

In low illumination situations, e.g., round 50 lux and below (as onemight find at a pub or low-light home environment) the blue channel maybe noisy. Such noise may outweigh or interfere with any watermark signalinformation in this color channel. Thus, regardless of the type of colortemperature, we can deemphasize signal contribution from the bluechannel. One example is:Gray=0.5*red−0.5*green+128The term “Gray” represents grayscale information at a particular imageor video location or pixel. In this equation, the grayscale value perlocation or pixel is 8-bits, but the techniques are not limited to this.The detector receives such grayscale information over an image area orover the entire image or video frame, and operates on such collectiveinformation to detect the encoded signal therefrom. In this example, thedetector operates on grayscale information representing portions of thered color channel (e.g., per location or pixel) and green color channel(e.g., per location or pixel), but not the blue color channel. The “128”in the above equation is used as a normalizing value to maintain an8-bit grayscale value. Otherwise, the resulting value may be above 255or below 0 (e.g., exceed an 8-bit number).

For high illumination situations, e.g., around 260 lux and above (as onemight find at an office), the noise in the blue channel is less.Moreover, the color channels can be weighted for detection in a mannerroughly proportional to the light in the various color channels and takeadvantage of the signal information from the blue channel. One exampleis:Gray=0.19*red−0.5*green+0.31*blue+128Like above, the term “Gray” here represents grayscale information at aparticular location or pixel. This equation can be used for eachlocation (or a portion of locations) in captured imagery. In thisexample, the detector operates on grayscale information representingportions of the red color channel (e.g., per location or pixel), greencolor channel (e.g., per location or pixel), and blue color channel(e.g., per location or pixel) in a manner weighted roughly according tolight distribution. Of course, and as with the other equations,different weighting coefficients may be used to match or coincide withparticular lighting sources. Thus, these weights are exemplary andshould not limit the scope of the disclosure.

In these high and low light situations, light intensity is moreimportant than color temperature. Things get a bit trickier whenconsidering mid level illumination.

For intermediate light levels, example detection weightings may include:Medium low (˜120 lux) gray=0.29*red−0.5*green+0.21*blue+128−CWF.gray=0.5*red−0.5*green+128−Incandescent.Medium high (˜190 lux) gray=0.39*red−0.5*green+0.11*blue+128.

To improve detection under adverse lighting conditions one can optimizeboth embedding and detection for the expected lighting conditions. Forexample, to be more robust under incandescent lighting, the embeddingcan be adjusted to put more signal in the red channel, where most of theincandescent light energy is. In one embedding example, a signal isembedded in a color image or video with a 1:1 embedding ratio across theblue/yellow:red/green channels so that the signal energy is more evenlyprovided across red, green and blue channels. In another embeddingexample, a signal is embedded in a color image or video with a 1:2embedding ratio across the blue/yellow channel and red/green channel sothat the signal energy is more weighted to the red/green channel. Otherratios can be determined according to particular lightingcharacteristics.

Detection rates of a 1:1 embedding ratio versus a 2:1 embedding ratio(blue/yellow:red/green channels) under incandescent low lighting isshown in FIG. 3 (using the same detection color channel weightings). Asshown, 1:1 embedding can improve detection rates.

Despite an even (or even a red/green biased) embedding, underincandescent lighting the blue channel is still noisy due to lowillumination, and detection may be compromised. We can furtherdeemphasize impact of this noisy blue channel by only detecting thesignal from the red and green channel. In one example, a grayscaledetector operates on red minus green information as shown below:Gray=0.5*red−0.5*green+128

The term “Gray” represents grayscale information at a particularlocation or pixel. This equation can be used for each location (or aportion of locations) in captured imagery. In this example, the detectoroperates on grayscale information representing portions of the red colorchannel (e.g., per location or pixel) and green color channel (e.g., perlocation or pixel).

Other color channel weightings can be used as well. For example, FIG. 4is a diagram showing detection rates with incandescent low lighting,using a 1:1 embedding ratio, and different color channel weightings. Thefollowing color channel detector weights are used in FIG. 4:

RED GREEN BLUE .19 −.5 .31 .3 −.5 .2 .4 −.5 .1 .5 −.5 0

As seen in FIG. 4, more favorable detection rates are achieved as theblue channel is deemphasized under Incandescent low light. At an imagingcapture distance beyond about 3.2 inches, the 0.1 and 0 weights for theblue channel yield better detection rates. The above weightings areprovided by way of example, and are not intended to limit the scope ofthis disclosure.

As a comparison, in higher lighting conditions and under CWF lighting,better detection rates beyond about 3.2 inches are achieved when theblue channel is included and not deemphasized as in FIG. 4. Thefollowing color channel detector weights are used in FIG. 5. (Thefollowing weightings are provided by way of example, and are notintended to limit the scope of this disclosure)

RED GREEN BLUE .19 −.5 .31 .3 −.5 .2 .4 −.5 .1 .5 −.5 0

With reference to FIG. 6, a signal detector may adapt its detectionprocess based on lighting information to optimize detection. Forexample, the detector can change how it operates on image data (e.g.,changing detection color channel weightings, using different detectionalgorithms, deemphasizing input from certain color channels, etc.) basedon information pertaining to lighting. Lighting information can bedetermined in a number of ways. For example, a user may be prompted toenter the type of lighting via a UI on a cell phone. The UI may presentpredetermined choices (e.g., outdoors, indoors, incandescent lighting,cool white light, etc.), e.g., on a touch screen for user selection.

A GPS or location module may be used to determine whether the cell phone(or other device) is located indoors or outdoors. For example, GPScoordinates can be provided to, e.g., Google maps or otherlocation/mapping service. The cell phone (or service) may use the GPScoordinates to determine whether they overlap or correspond to astructure, building or outdoors. This information (e.g., outdoors) maybeused to determine the type of lighting information (e.g., daylight). Ifa time indicator indicates nighttime (e.g., dark), the process can beconfigured to provide lighting information associated with a cell phonecamera flash.

Another option is for the signal detector to receive lightinginformation from a camera on the cell phone, e.g., the auto-whitebalance algorithm in the camera. The auto-white balance is associatedwith “color temperatures,” a way of quantifying the color of light. Suchcolor temperature information can be used to determine the type oflighting or lighting information. For example, a predetermined autowhite balance value (or range of values) can be used to indicate thatthe current lighting source is more likely to correspond to, e.g., redlight (e.g., more likely incandescent lighting).

Another method may examine image statistics (e.g., using imagehistograms) associated with captured imagery. For example, a magnitudeof high frequency noise levels in the blue channel can be analyzed todetermine the type of lighting. In this image histogram example, apredetermined noise level in the blue channel may be used to indicateincandescent lighting; or a noise level below such predetermined levelmay be used to indicate CWF lighting. Intermediate noise levels may beused to indicate intermediate lighting.

Analyzing image statistics may also be used to determine differentlighting regions within a captured image or video. For example, afteranalyzing image statistics, an image may be determined to predominatelycorrespond to CWF lighting. However, the image statistics may identifyregions within the image that may include shadows or other lightingissues. These statistics can be used by the detector to use a firstdetection process for the majority of the image, and a second detectionprocess for the shadow (or different lighting conditions) areas.

Analyzing image statistics may also include an analysis of a ratio(s) ofone color channel to other color channels (e.g., Blue vs. Red; Blue vs.Red/Green; Blue vs. Green; Green vs. Red, and/or so on). One way toestablish a ratio(s) is to find minimum points, maximum points andquartile points in the color channels (e.g., histograms can be used todetermine such). Ratios can be determined e.g., during color conversion.A detector can be trained to recognize certain types of lighting basedon a given ratios. For example, a detector can be trained against a setof captured color images or video. The image set would preferably havevaried color biases (e.g., red, blue, green, black, etc.), and becaptured across different lighting conditions (e.g., low light, regularlight, incandescent lighting, CWF, sunlight, black light, coloredlights, etc.). Ratios can be matched to known lighting conditions(and/or known image content), and color weightings can be determined forthose ratios. Once this ratio (and corresponding color weighting)information is collected during training, the detector can assignpredetermined color weightings going forward based on determined colorchannel ratios.

Additionally, a cell phone or other imaging device may also include orcommunicate with a light meter. The light meter may provide informationregarding the light level (e.g., light intensity). This light levelinformation may be used as lighting information to adapt a detectionprocess or signal detector.

Returning to FIG. 6, in one implementation, the signal detector useslighting information to select a color channel weighting. For example,if the lighting information indicates that the lighting source is morelikely to be CWF, under regular lighting levels, the detector may selectand use the Gray=0.19*red−0.5*green+0.31*blue+128 weightings (or otherweighting determined for this type of lighting). The detector may usethe lighting information to look up weightings in a table, registry ordatabase. Or, the detector may be preprogrammed to use certainweightings based on the lighting information. If a signal is detected bythe detector, the corresponding message or payload may be output orcommunicated (e.g., to another process in the cell phone, for display tothe user, to a remotely located device, to a home or office network,etc.). If the signal is not detected, the detector may, optionally,decide to adjust the weightings and try detection again.

In accordance with other aspects of the present disclosure, shadows areaddressed.

Often, when a smart phone camera is used to capture imagery of awatermarked object, or to capture imagery to perform a so-calledfingerprinting process, the smart phone casts a shadow on the object.Such a situation is shown in FIG. 7. A smart phone 202 includes a camerasystem (indicated by the position of a lens 204 on the back side of thephone) that captures image data from a rectangular area 206. The smartphone 204 blocks some of the light illuminating the area 206, casting ashadow 210.

Although shadow 210 is shown as uniform, typically it varies indarkness—becoming less distinct at the outer edges, particularly withdiffuse lighting.

Shadow 210 can be detected, and addressed, in various ways.

One way to detect the shadow is to analyze the captured image data for acontour that mimics, in part, the profile of the cell phone. Edgefinding algorithms are well known, and can be applied in thisapplication. Aiding identification of the shadow is the fact that edgesof the shadow are usually parallel with edges of the captured imageframe (at least for generally rectangular smart phones, held parallel tothe imaged area 206). Once candidate edges are found in the image data,they can be matched against a series of reference templatescorresponding to shadows produced by different edges of the smart phone,and under different lighting conditions, to identify the shadow edge.

To reduce confusion with other subjects within the camera's view, theedge identification can be conducted by analyzing luminance channeldata—disregarding color information.

Another technique for shadow identification briefly strobes the scenewith a flash from the smart phone camera (e.g., by an LED light directedinto the camera's field of view). An image frame without the strobe iscompared with an image frame that is illuminated with the extralighting—analyzed for edges found in the former that are missing in thelatter. Such edges correspond to shadows cast by the phone.

Another way of identifying the shadow 210 exploits the fact that, if thephone is moved, the shadow's position within the field of view isgenerally stationary, whereas the subject being imaged apparently moves.This is illustrated in FIGS. 8a and 8b . In these drawings the cameracaptures imagery from a field of view area 206 including a piece ofpaper 212. The paper is printed with text.

As the user positions the camera to frame the desired shot (or as thecamera is slightly jittered by the user's hand in normal use), theapparent position of the piece of paper 212 within the field of view 206moves. However, the shadow 210 is essentially fixed in the frame (sincethe camera casting the shadow moves with the field of view). Byanalyzing two or more frames, the image data can be resolved into twocomponents: image features that change position (e.g., features from theprinted paper 212) and image features that are apparently static (e.g.,the shadow edges).

In the example illustrated, it can be seen that the shadow 210 of thesmart phone in FIG. 8a encompasses the “e” in “The,” and the “e” in“jumped.” In FIG. 8b , the phone has been pointed a bit more to theright (indicated by displacement 214), so the left edge of the paper isdepicted nearly at the left edge of the image frame. The shadow nolonger encompasses the “e” s. The shadow has stayed stationary; thesubject imagery has appeared to move.

By spatially correlating frames of image data captured at differentinstants, the static features can readily be identified (e.g., theshadow boundary). Again, such operation is desirably performed inluminance data, so to reduce confusion with other features.

Once the shadow edges have been identified (by the foregoing, or othertechniques), subsequent processing can mitigate the effects of theshadow.

One approach is simply to recognize that the shadowed region is inferiorin image quality, and to disregard it—where possible. Thus, for example,in ranking candidate pixel regions for submission to a watermarkdetector, the shadowed regions may be discounted.

Another approach is to compensate the captured imagery to redress theshadowing effect.

One way to do this is to estimate the reduction in subject luminancecaused by the shadow at different points in the image frame, and thenadjust the luminance across the image reciprocally. This can be done byexploiting the fact that natural imagery is highly spatially correlated.(If one pixel is purple in color, then the probability that a nearbypixel is also purple in color is much greater than would occur withrandom chance alone.)

Consider the example of FIGS. 8a and 8b . The substrate of paper 212 hasa background color, which is reflected in captured image pixel values.In an LAB or other dual chrominance space representation, the color datacorresponding to these pixels depicting the paper background isinvariant with illumination; chrominance does not change with shadowing.

To compensate the imagery for the shadow, the image is analyzed for oneor more spatially close pairs of pixel regions—one falling inside theshadowed region, and one falling outside—with similar color values. Thepaper substrate is an example. The method assumes that where spatiallyclose regions are also close in chrominance values, that they form partof a common object (or similar objects) within the camera's field ofview. If they form part of a common or similar object, and are similarin chrominance, then the difference in luminance is a measure of theshadow's darkness at the shadowed of the two regions.

By examining the luminance at different pixels representing the paper212, the luminance profile of the shadow—at least in the region of thepaper 212—can be determined. Likewise with other regions of similarchrominance found on both sides of the shadow boundary. A complementaryluminance correction can then be applied—brightening the pixels in theshadowed region to match the luminance of similarly-colored pixels thatare nearby yet unobscured by shadow. (Darkening the pixels outside theshadow is also a possibility.)

Spatial proximity of similarly-colored regions is desired, but is notessential. The same technique is applicable even if similarly-coloredregions are found at opposite edges of the image frame.

It will be recognized that many environments have different illuminationsources. A typical scenario is an interior space with exterior windows,which also has overhead incandescent or fluorescent lighting. Thenatural lighting through the windows provides a spectrum different thanthe artificial lighting. The shadowing caused by the smart phonetypically blocks one light source (e.g., the artificial lighting) morethan the other.

By assessing the image chrominance in a shadowed region versus in anun-shadowed region, information about the ambient lighting can bediscerned, and further compensation may thus be applied.

Consider again the examples of FIGS. 8a and 8b . The shadow 210 mayblock overhead fluorescent illumination, causing the right part of thepaper to be illuminated exclusively with natural daylight. The leftedge, in contrast, is lit with both daylight and fluorescentillumination. These different illuminations can cause the apparentchrominance of the paper to vary from the left edge to the right edge.

Despite the variance in apparent chrominance across its length, analysisof other image features can still indicate that the region of pixelsspanning the paper substrate likely corresponds to a unitary object.This can be confirmed by edge analysis (finding the outer boundingrectangle). Texture analysis can also be employed to determine that thedepicted item has generally homogenous image texture within thediscerned edge boundary. Similarity in chrominance (although with alarger tolerance, reflecting the different illumination) can also beused.

Once regions of a common object—some in the shadow and some out of theshadow—are identified, the chrominance of such regions can be compared.The discerned difference is likely due to absence of one light source inthe shadowed region. By assessing this difference, a chrominancecorrection can be applied, e.g., so that the left and right edges ofdepicted paper substrate 212 have the same chrominance values.

Shadows can also be used as a gross measure of proximity of the cellphone camera to the object being imaged. The darker the shadow (and/or,the more well-defined the shadow boundary), the closer the camera is tothe subject. If analysis of a temporal sequence of image frames showsthat a shadow is becoming darker, or more distinct, the phone can inferthat the camera is being moved closer to the subject, and then knows,e.g., in what direction a focus control should be adjusted.

The computing environments used to implement the above processes andsystem components encompass a broad range from general purpose,programmable computing devices to specialized circuitry, and devicesincluding a combination of both. The processes and system components maybe implemented as instructions for computing devices, including generalpurpose processor instructions for a variety of programmable processors,including microprocessors, Digital Signal Processors, etc. Theseinstructions may be implemented as software, firmware, etc. Theseinstructions can also be converted to various forms of processorcircuitry, including programmable logic devices, application specificcircuits, including digital, analog and mixed analog/digital circuitry.Execution of the instructions can be distributed among processors and/ormade parallel across processors within a device or across a network ofdevices. Transformation of content signal data may also be distributedamong different processor and memory devices.

The computing devices used for signal detection and embedding mayinclude, e.g., one or more processors, one or more memories (includingcomputer readable media), input devices, output devices, andcommunication among these components (in some cases referred to as abus). For software/firmware, instructions are read from computerreadable media, such as optical, electronic or magnetic storage mediavia a communication bus, interface circuit or network and executed onone or more processors.

The above processing of content signals may include transforming ofthese signals in various physical forms. Images and video (forms ofelectromagnetic waves traveling through physical space and depictingphysical objects) may be captured from physical objects using cameras orother capture equipment, or be generated by a computing device. Whilethese signals are typically processed in electronic and digital form toimplement the components and processes described above, they may also becaptured, processed, transferred and stored in other physical forms,including electronic, optical, magnetic and electromagnetic wave forms.The content signals can be transformed during processing to computesignatures, including various data structure representations of thesignatures as explained above. In turn, the data structure signals inmemory can be transformed for manipulation during searching, sorting,reading, writing and retrieval. The signals can be also transformed forcapture, transfer, storage, and output via display or audio transducer(e.g., speakers).

While reference has been made to cell phones, it will be recognized thatthis technology finds utility with all manner of devices—both portableand fixed. PDAs, organizers, portable music players, desktop and laptopcomputers, tablets, pads, wearable computers, servers, etc., can allmake use of the principles detailed herein. Particularly contemplatedcell phones include the Apple iPhone, and cell phones following Google'sAndroid specification (e.g., the G1 phone, manufactured for T-Mobile byHTC Corp.). The term “cell phone” should be construed to encompass allsuch devices, even those that are not strictly-speaking cellular, nortelephones.

(Details of the iPhone, including its touch interface, are provided inpublished patent application 20080174570. This published application ishereby incorporated by reference in its entirety.)

The design of cell phones and other computers that can be employed topractice the methods of the present disclosure are familiar to theartisan. In general terms, each includes one or more processors, one ormore memories (e.g. RAM), storage (e.g., a disk or flash memory), a userinterface (which may include, e.g., a keypad, a TFT LCD or OLED displayscreen, touch or other gesture sensors, a camera or other opticalsensor, a microphone, etc., together with software instructions forproviding a graphical user interface), a battery, and an interface forcommunicating with other devices (which may be wireless, such as GSM,CDMA, W-CDMA, CDMA2000, TDMA, EV-DO, HSDPA, WiFi, WiMax, or Bluetooth,and/or wired, such as through an Ethernet local area network, a T-1internet connection, etc). An exemplary cell phone that can be used topractice part or all of the detailed arrangements is shown in FIG. 1,discussed above. The processor can be a special purpose electronichardware device, or may be implemented by a programmable electronicdevice executing software instructions read from a memory or storage, orby combinations thereof. (The ARM series of CPUs, using a 32-bit RISCarchitecture developed by Arm, Limited, is used in many cell phones.)References to “processor” should thus be understood to refer tofunctionality, rather than any particular form of implementation.

In addition to implementation by dedicated hardware, orsoftware-controlled programmable hardware, the processor can alsocomprise a field programmable gate array, such as the Xilinx Virtexseries device. Alternatively the processor may include one or moreelectronic digital signal processing cores, such as Texas InstrumentsTMS320 series devices.

Software instructions for implementing the detailed functionality can bereadily authored by artisans, from the descriptions provided herein,conclusions, and other determinations noted above.

Typically, devices for practicing the detailed methods include operatingsystem software that provides interfaces to hardware devices and generalpurpose functions, and also include application software that can beselectively invoked to perform particular tasks desired by a user. Knownbrowser software, communications software, and media processing softwarecan be adapted for uses detailed herein. Some embodiments may beimplemented as embedded systems—a special purpose computer system inwhich the operating system software and the application software isindistinguishable to the user (e.g., as is commonly the case in basiccell phones). The functionality detailed in this specification can beimplemented in operating system software, application software and/or asembedded system software.

Different of the functionality can be implemented on different devices.For example, in a system in which a cell phone communicates with aserver at a remote service provider, different tasks can be performedexclusively by one device or the other, or execution can be distributedbetween the devices. Thus, it should be understood that description ofan operation as being performed by a particular device (e.g., a cellphone) is not limiting but exemplary; performance of the operation byanother device (e.g., a remote server), or shared between devices, isalso expressly contemplated. (Moreover, more than two devices maycommonly be employed. E.g., a service provider may refer some tasks,functions or operations, to servers dedicated to such tasks.)

In like fashion, data can be stored anywhere: local device, remotedevice, in the cloud, distributed, etc.

Operations need not be performed exclusively byspecifically-identifiable hardware. Rather, some operations can bereferred out to other services (e.g., cloud computing), which attend totheir execution by still further, generally anonymous, systems. Suchdistributed systems can be large scale (e.g., involving computingresources around the globe), or local (e.g., as when a portable deviceidentifies nearby devices through Bluetooth communication, and involvesone or more of the nearby devices in an operation.) For example, a cellphone may distribute some or all of the image data and/or lightinginformation to the cloud for analysis, e.g., to detect an encoded signalor to determine image statistics. A detection result, a partial resultor computation stages may be communicated back to the cell phone forreview or further computation or actions.

Concluding Remarks

Having described and illustrated the principles of the technology withreference to specific implementations, it will be recognized that thetechnology can be implemented in many other, different, forms. Toprovide a comprehensive disclosure without unduly lengthening thespecification, each of the above referenced patent documents is herebyincorporated by reference in its entirety.

While the above application discusses consideration of two lightingconditions during a detection process, e.g., lighting level (e.g., luxlevel) and color temperature (e.g., type of lighting), the presentdisclosure is not so limited. For example, a detector may only considerone of these considerations when determining color channel weightings.

The particular combinations of elements and features in theabove-detailed embodiments are exemplary only; the interchanging andsubstitution of these teachings with other teachings in this and theincorporated-by-reference patent documents are also contemplated.

What is claimed is:
 1. A non-transitory computer readable mediumcomprising instructions stored thereon that when executed by one or moreprocessors, cause a portable device comprising a camera for capturingdata representing imagery or video, a radio frequency (RF) transceiver,a touchscreen display, a graphical user interface for providing a userinterface on the touchscreen display, to perform the following: obtaindata representing imagery or video provided by the camera; obtainlighting information via the graphical user interface, the lightinginformation associated with the captured data representing imagery orvideo; adapt an encoded signal detector based on the lightinginformation, in which the adapted encoded signal detector is configuredto detect whether a signal is encoded within the data representingimagery or video.
 2. The non-transitory computer readable medium ofclaim 1 in which the adapt instructions comprise instructions that applydifferent weightings to different color channels of the datarepresenting imagery or video for detection, the different weightingsbeing associated with the lighting information.
 3. The non-transitorycomputer readable medium of claim 2 in which the lighting information isassociated with incandescent lighting, and the weightings deemphasize ablue color channel.
 4. The non-transitory computer readable medium ofclaim 3 in which the blue color channel is deemphasized to the point ofnot being used in detecting the encoded signal.
 5. The non-transitorycomputer readable medium of claim 2 in which the lighting information isassociated with daylight or cool white fluorescent lighting, and theweightings are applied across red, green and blue channels.
 6. Thenon-transitory computer readable medium of claim 1 in which the adaptedencoded signal detector operates on a grayscale representation of thedata representing imagery or video.
 7. The non-transitory computerreadable medium of claim 1 in which the encoded signal detectorcomprises a digital watermark detector.
 8. A portable apparatuscomprising: a camera for capturing data representing imagery or video; atouchscreen display; means for communicating and receiving data; meansfor obtaining data representing imagery or video, the data representingimagery or video having been captured with said camera; means forobtaining lighting information associated with image capture of the datarepresenting imagery or video, in which the lighting information isassociated with a light level or color temperature; and means forprocessing the data representing imagery or video to determine whether asignal is encoded therein, in which said means for processing utilizesobtained lighting information to determine whether the signal is encodedin the data representing imagery or video.
 9. The portable apparatus ofclaim 8 in which the lighting information is associated with the lightlevel and the color temperature.
 10. The portable apparatus of claim 8in which the color temperature is associated with incandescent lighting.11. The portable apparatus of claim 9 in which the color temperature isassociated with incandescent lighting.
 12. The portable device of claim8 in which the data representing imagery or video comprises a pluralityof color channels including at least a blue channel, and in which theblue channel is deemphasized by said means for processing.
 13. Theportable device of claim 9 in which the data representing imagery orvideo comprises a plurality of color channels including at least a bluechannel, and in which the blue channel is deemphasized by said means forprocessing.
 14. The portable apparatus of claim 8 in which the colortemperature is associated with daylight or cool white fluorescentlighting.
 15. The portable apparatus of claim 8 in which said means forprocessing processes a grayscale representation of the data.
 16. Theportable apparatus of claim 8 in which the data representing imagery orvideo comprises two color channels, in which the signal comprises awatermark signal, with the watermark signal embedded in a first colorchannel, and the watermark signal embedded in a second color channelwith a signal polarity that is inversely related to a signal polarity ofthe watermark signal in the first color channel.
 17. The portableapparatus of claim 8 in which the portable apparatus comprises a cellphone.
 18. A portable apparatus comprising: a camera for capturing datarepresenting imagery or video; a touchscreen display; a radio frequency(RF) transceiver for transmitting and receiving data; means forobtaining lighting information associated with capturing of the datarepresenting imagery or video by said camera; means for extractingfingerprint information from the data representing imagery or video, inwhich said means for extracting utilizes obtained lighting information.19. The portable apparatus of claim 18 in which said means forextracting applies different weightings to different color channels ofthe data representing imagery or video, the different weightings beingassociated with the obtained lighting information, and in which theweightings deemphasize the blue color channel.
 20. The portableapparatus of claim 18 in which the obtained lighting information isassociated with incandescent lighting.